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-ABSTRACT 



We present the results of a i^fonte Carlo Study of the power 
of a new multivariate goodness -of- fit test. 



INTRODUCTION 

Foutz [1] developed a new test of goodness -of- fit for multi- 
variate distributions. The test can also be used to fit univariate 
distributions; in an earlier paper [2] the authors compared the 
Foutz test with the Chi-square and Kolmogorov-Smimov test. Tlie 
results indicated that the Foutz test is more powerful in detecting 
certain characteristics than the other two tests. This paper deals 
with the performance of the test when fitting multivariate distri- 
butions. Nbre specifically we investigate the power of the test 
when fitting bivariate and trivariate normal distributions for 
various choices of the mean vector and the covariance matrix. In 
the second section we present a brief description of the Foutz 
test; a discussion of the simulation procedure is in the third sec- 
tion and the results of the simulation are in the final section. 



1 



THE FOUTZ TEST 



Let X^, be a random sample of site n-1 from a 

p-variate continuous distribution. The first step of the Foutz 

test is to divide the sample space into n statistically equivalent 

blocks and then determine a "continuous empirical 

distribution function (CEDF)" H . The test statistic for the hypo- 

n. 

thesis that the true c-d-f is H is 



F 

n 



Sup 

all events 
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P„C3) - P„(3) 



( 1 ) 



where Pj^(8) and P[^(8) are the empirical and hypothesized probabi- 
lity measures of an event 8, confuted from and H respectively. 
An equivalent computational formula for F^ is 



r 

■ n 



n 

I 

i=l 



max 



[0,- - D.] 
’ n i-* 



where = P[X £ 8 JH] . 

A general procedure for the construction of statistically equi- 
valent blocks is the following. Select n-1 "cutting functions" 
h|^(X) , k = 1, 2,..., n-1, such that h^CiO has a continuous distri- 
bution, and a permutation k^ ,k., , . . . ,k^_^ of the integers 1,2,..., n-1. 
Order the samples X- according to the value of h, (X.); let X(k^) be 

1 1 X 

the vector associated with the k^th order statistic. Partition the 

sample space into two blocks B, and defined by 

i. L 

B^={X: hj^^(.X) < [X(k^)]} and B 2 = B^ . 

At the second step, if k 2 < k^ , order the k^ sample vectors in B^ 
according to (X) and let X(k 2 ) be the k^th order statistic. Par- 



tition B^ into two sub-blocks B^^ and B^^; at this stage the sample 
space is partitioned into three blocks B^^, B^^^ and B^g = B-,. If 
k-, > kp order the n - 1 - k^ sample values in the block B., 
according to h^^(X). Let XCk^) be the (k 7 - k^)th order statistic 



and partition the block B, into two sub-blocks and B.,-,; take 
B^O “ Continue the process until all the cutting functions are 

exhausted; this results in a partition of the sample space into n 
statistically equivalent blocks 3^, 3-,,..., 6^^. Nfore details on 
the procedure and some examples are available in [3] . 

The null c*d*f of the test statistic (necessar)’ to deter- 
mine the critical values) is quite difficult to derive even for 
small n; for n = 3, 4, 5 formulas for the exact c-d*f are in [2]. 
Foutz proposed a large sample normal approximation with mean 
w = e'^ and = (2e'^ - 5e'^)/n. In our earlier study [2] we found 
that with this approximation the observed significance level is 
about 10-20^ smaller than the nominal values for sample size n - 1 = 
20, 30, 50. We therefore proposed the use of empirically generated 
(based on 30,000 simulated F^ values) critical points in Table I 
below. 

TABLE I. EMPIRICAL CRITICAL VALUES FOR FOUTZ TEST 



Sample Size 


20 


30 


50 


Significance level 


.10 


.42714 


.41903 


.40816 


.05 


.44865 


. 43533 


.42116 


.01 


.48659 


.46579 


.44487 



We have also generated a variation of the large sample nomal 
approximation to calculate more accurate estimates of the critical 
values. We are presently in the process of using this approxima- 
tion to generate tables for a spectrum of critical values and 
sample sizes. 

SIMULATION PROCEDURE 

To investigate the power of the Foutz test 5,000 replicates 
each of samples of size 20 from several bivariate and trivariate 
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normal distributions were generated; in all cases the hypothesis 
tested was that the samples are from a bivariate/trivariate normal 
distribution with zero mean vector and covariance matrix the Iden- 
tity. The true values of the means, variances and covariances for 
the generated samples were chosen so as to study the effect of 
(i) shifts in the means only (ii) shifts in the variances only 
(iii) shifts in covariances only and few cases involving a combina- 
tion of all three. 

The method of blocking we implemented was as follows. We let 
the samples themselves implicitly determine the permutation k^, k^, 

. . . , k^_j^ and the order of the cutting functions which were all 
taken to be coordinate functions i.e. , the i^^ coordi- 

nate of the sample vector X, for some i. ^The following example 
with p = 2 (bivariate samples) will illustrate the procedure. Sup- 
pose x^, x^, ..., ^ are the observed sample vectors. The first 
cut on the p-dimensional sample space is made at x^^^ the first 
coordinate of the first sample vector x^. This partitions the sam- 
ple space into two blocks defined by 



1^^ coordinate 



2^^ coordinate 



(-00 

(xW 






(-0 

(-0 



+ 00 ^ 

4 * 00 ^ 



The second sample vector will be in one of the two blocks 
Bj^ or B^. .Assume that it is in B.,; partition B., into two sub-blocks 

1^^ coordinate 2^*^ coordinate 



21 


(x)'’ , 




'22 


(xj^\ +oo) 


(Xp\ +oo) 



4 



( 7 ) 

where is the value of the second coordinate of the sample 

vector x^. At this stage the sample space is partitioned into 
three blocks B and B^^. Continuing this process (letting 

the cutting function at the q stage to be ^ ^ the r^^ coordinate 
of the X where r = [ (q-1) mod p + 1] ) ujitil all the sample vec- 

tors are exhausted will result in a partition of the sample space 
into n statistically equivalent blocks. In our simulation we 
used both rectangular and polar/ spherical coordinates to examine 
if one scheme is more adept at detecting certain types of violations 
of the null hypothesis. IVhen using spherical coordinates the first 
coordinate is taken to be the radius vector (same for polar coordi- 
nates) , the second coordinate is the angle in the horizontal plane 
and the third coordinate is the angle measured from the vertical axis. 
Figures 1 and 2 granhically demonstrate the constmaction of the 
blocks for five bivariate samples using the rectangular and polar 
coordinate systems. The implicit permutation and the order of the 
coordinate cutting "unctions are also included in the figures. It 
should be noted tha', for polar/ spherical coordinates it is necessary 
to make an additional initial cut, which we took at 9 = 0°. 

The probability contents of the statistically equivalent blocks 
under the null hypothesis (multivariate normal with Zero mean vector 
and covariance matrix the Identity) are easily computable as products 
of univariate probabilities (normal, Chi square and uniform) for the 
rectangular coordinate system as well as the polar/spherical coordi- 
nate system. 

PESULTS 

For ease of comprehension, the results for shifts of m.ean or 
variance/covariance are given as power curves in Figures 3-7. 

All power curves are based on 5,000 replications at the .05 signi- 
ficance level. The results are indicative of those obtained for 
significance levels of .01 and .1. Full details are available in 
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Linhart [3] . 

Shifts in mean are detected well. Figures 3 and 4 show that a 
shift of one standard deviation in the mean results in about a 60% 
rejection rate for both bivariate and trivariate sauries. The 
rectangular method of blocking consistently resulted in a higher 
rejection rate than did the polar/ spherical method. 

Power curves for shifts in variance are given in Figures 5 and 

6. Small shifts in variance are not detected very well, but larger 
shifts and shifts in more than one component resulted in higher 
rejection rates. Neither blocking method produced rejection rates 
significantly better than the other except in the trivariate case 
vdien one variance was shifted. In general, the polar/ spherical 
method of blocking gives slightly higher rejection rates, but not 
consistently. 

The power curves for shifts in covariance are given in Figure 

7. Except for highly correlated data, neither blocking scheme 
detects these shifts very well. The polar/ spherical method of 
blocking usually gives somewhat higher rejection rates than the 
rectangular method. 

To test for possible confounding in detection of shifts in 
both mean and variance/covariance, we made the series of runs 
listed in Tables II and III. Shifts are generally of increasing 
amount as one looks down and to the right in the tables. It is 
also true that rejection rates increase as one looks down and to 
the right, indicating there is no detectable confounding. The 
rectangular method of blocking generally gave better results with 
multiple shifts. 

All of the above results are based on a sample size of 20 and 
significance level of .05. To determine possible trc=*nds over sample 
size or significance level we made the series of runs given in 
Tables IV and V. Larger sample sizes generally led to higher rejec- 
tion rates, as is to be anticipated. Violations of this did occur 
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when rejection rates were close to the significance level, i.e., 
the shift was not detected very well. The rectangular blocking 
method generally gave higher rejection rates. 

In conclusion, the Foutz test is adept at detecting shifts in 
mean, less powerful at detecting shifts in variance, and poor in 
detecting correlated variates unless they are highly correlated. 

In addition, shifts in mean are not disguised when a shift in 
variance/covariance is also present. 



7 



BIBLIOGRAPHY 



Foutz, R.V. , "A Test for Goodness -of -Fit Based on an Enpirical 
Probability Measure", .^als of Statistics 8 (1980), 989 - 
1001. 

Franke, R. , 8 Jayachandran , T. , "An Empirical Investigation of the 
Properties of a .New Goodness-of-Fit Test", to appear in the 
Journal of Statistical Computation and Simulation. 

Linhart, R.J. , "A Study of Foutz's Multivariate Goodness-of-Fit 
Test", M.S. Thesis, Naval Postgraduate School, Monterev, 

CA 93940, March 1982. 



8 




9 



Ki:t:TAN(;iiij\R (;(X)ri)1natl blockinc i-'h;. i: i’olar ccwRi)iN,vn-; bl(k:king 



I^ECriQN RATE rCJECflON RATE 



1.0 



0.8 



0.6 



0.4 



0.2 



Polar 




0.0 



-3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 



Y 




Y 

FIGURE 3: POWER CURVES FOR SHIFTS IN MEAN 



10 




ION RATE REJECTICW RATE 



1.0 



0.8 ‘ 

I 

1 

0.6 1 



0.2 

1 

O.O3L 



1.0 I 

t 

0.8 t 

♦ 

0.6 . 

1 

i * 

0.2 - 
0.0 

-3 

FIGURE 




Y 




Y 

4 : POWER CURVES FOR SHIFTS IN MEAN 



11 



l^iJECTiaN RA'l’E RE.TECTIO'J RATE 



1.0 





a 



FIGURE 5: POWER CURVES FOR SHIFTS IN VARIANCE 
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REJECTION RATE REJECTION RATE 





FIGURE 6: POWER CURVES FOR SHIFTS IN VARIANCE 
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rejection ra'it; 
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FIGURE 7: POWER CURVES FOR SHIFTS IN COVARIANCE 
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TABLE II 



l^EJECTION R^TES FOR MULTIPLE SHIFTS IN 
^EAN .AND V.ARIANCE -COVARIANCE 




Mean 




.0488 

.0482 

.1710 

.1388 

.5606 

,4348 

.9418 

.8576 

.9998 

.9950 



.0912 

.1176 

.2398 

.2482 

.7346 

.5952 

.8774 

,8588 

.9998 

.9990 



.1986 

.1522 

.3110 

.2498 

.6384 

.5334 

.9350 

.8722 

.9902 

.9772 



.2500 

.2162 

,3702 

.3402 

.6828 

.6316 

.8658 

,8308 

.9950 

.9882 



.9658 

.9572 

.9720 

.9650 

.9820 

.9764 

.9892 

.9840 

.9990 

.9964 



* First Entry - Rectangular coordinate blocking 
Second Entrv’’ - Polar coordinate blocking 
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TABLE III 



REJECTION R.MES FOR MULTIPLE SHIFTS IN 
NE.AN AND VARIANCE -COV,AILANCE 

a = .05, n = 21* 

Sigma /I 0 0\ / 1 0 .6\ / 5 0 0\ / 10 0 .95\ 

010 01 0 010 0 1 0 ) 
\0 0 1/ \.6 0 1/ \0 0 1/ \.95 0 1 / 

Mean 




.0440 


.0674 


.0518 


.0582 


.0480 


.1830 


.0280 


.1176 


.3728 


.6352 


.1174 


.2912 


.7400 


.9074 


.7392 


.7454 


.9982 


.9956 


.9726 


.9752 



.5392 


.:^828 


.4584 


.7840 


.5708 


.7946 


.5034 


.8020 


.6852 


.8206 


.6254 


.8422 


.9270 


.9668 


.8602 


.9454 


.9716 


.9736 


.9742 


.9774 



* First Entry - Rectangular coordinate blocking 
Second Entry - Polar coordinate blocking 
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5 0 0\ 

0 5 0 

0 0 5/ 



.9770 

.9720 



.9832 

.9740 



.9888 

.9824 



.9930 

.9870 



.9978 

.9976 



T.\BLE IV 





REJECTION 


RATES FOR INCREASLN'G SAWLE SIZES 




Sample siz 


e (n-1) 


20 


30 


50 


Shift 






ct = .01 




u = /.5 \ 




.0574 


.0860 


.1270 


1 0 / 




.0430 


.0564 


.0754 


u = /.5 \ 




.1294 


.2026 


.3652 


(.5 j 




.1230 


.1418 


.2508 


E = / 1 


.5 \ 


.0126 


.0140 


.0176 


1.3 


J 


.0136 


.0152 


.0170 


E = (1 


° ) 


.1864 


.2722 


.4522 


1 0 




.1578 


.2244 


.3744 








ct = .05 




( 1 ) 




.1710 


.2170 


.2914 




.1388 


.1630 


. 2238 


u = /.5\ 




.3024 


.4144 


.6030 


1.5 / 




.3164 


.3076 


.4826 


E = / 1 


.3 \ 


.0576 


.0624 


.0728 


1.3 


l) 


.0656 


.0624 


.0760 


E = /I 




, .3786 


.4884 


.6756 


\ 0 


3 / 


.3342 


.4304 


.6016 








a = .10 




u = /.5 \ 




.2700 


.3228 


.4256 


1 0 / 




.2298 


.2658 


.3400 


u = / .5 \ 




.4406 


.5424 


.7190 


1.5/ 




.4534 


. 4336 


.6066 


E = / 1 




.1173 


.1174 


.1396 


\ .5 


1 ) 


.1258 


.1196 


.1422 


E = /I 


0 ] 


.5150 


.6132 


.7800 


\ 0 


3 / 


.4640 


.5566 


.7160 



First Entry - Rectangular coordinate blocking 
Second Entry - Polar coordinate blocking 
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y 



RFJ12CTI0N R.ATES FOR INCREASING 3.AMPLH SIZES 



Sample 
Shift 
u = / 



size 


(n-l) 


20 


30 


50 








ct = .01 




.5 \ 




.0480 


.0680 


.1036 


0 ) 




.0280 


.0362 


.0526 


0 / 










.5 \ 




.1738 


.2932 


.5040 






.0848 


.1662 


.3428 


.5 / 










1 0 


.5\ 


.0106 


.0138 


.0148 


0 1 


0 


.0124 


.0134 


.0144 


.3 0 


1/ 








3 0 


o\ 


.1606 


.2054 


.3528 


0 1 


0 ) 


.0838 


.1138 


.2080 


0 0 


1/ 









u = 



u = 



z = 



a = .05 



5 ^ 






.1438 


.1974 


.2742 


0 






.1036 


.1256 


.1646 


0 y 
5 > 


1 




.3642 


.5118 


.7268 


5 






. 2198 


.3588 


.5868 


5 y 


1 










1 


0 


.3\ 


.0463 


.0588 


.0656 


0 


h 


0 ) 


.0512 


.0488 


.0540 


3 


0 


.1 / 








3 


0 


0 \ 


. 3438 


.3976 


.5734 


0 


1 


0 ) 


.207i 


.2708 


.4126 



ct = .10 



u = 



I = 



Z = 



,5 ^ 






.2484 


.3024 


.3876 


0 






.1874 


.2132 


.2650 


0 y 


f 










.5 \ 






.4948 


.6396 


.8272 


, 5 






.3584 


.4912 


.7040 


,5 J 












1 


0 


,3\ 


.0972 


.1142 


,1232 


0 


1 


0 


.1086 


.1030 


.1102 


,3 


0 


1 / 








3 


0 


0\ 


. 4736 


.5264 


.6880 


0 


1 


0 ) 


.3156 


.3880 


.5450 


0 


0 


1/ 









First Entry - Rectangular coordinate blocking 
Second Entry - Polar coordinate blocking 
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